in_node_exporter: more pluggable structure by nokute78 · Pull Request #8077 · fluent/fluent-bit

nokute78 · 2023-10-22T01:30:45Z

This patch is make more pluggable structure.
It is to remove similar codes for maintainability.

Define common collector struct flb_ne_collector
Add/rename common/platform specific source and header
- Add ne_filefd.[ch], ne_stat.[ch] and ne_vmstat.[ch]
- Rename ne_systemd.[ch] -> ne_systemd_linux.[ch].

struct flb_ne_collector

This struct represents common collector like plugin instance.

struct flb_ne_collector {
    const char *name;
    int coll_fd;
    int interval;
    int activated;

    /* callbacks */
    int (*cb_init) (struct flb_ne *ctx);
    int (*cb_update) (struct flb_input_instance *ins, struct flb_config *conf, void *in_context);
    int (*cb_exit) (struct flb_ne *ctx);

    struct mk_list _head;
};

Name	Description
name	Collector name. e.g. `cpufreq`
interval	Polling interval to collect metrics in second. If `collector.%name%.scrape_interval` is defined on config map, this value is set by the value of config map. Default : `scrape_interval`
coll_fd	File descriptor that is returned from `flb_input_set_collector_time`
activated	If enabled, this collector is enabled and collect metrics.
cb_init	Callback to initialize this collector. If it is NULL, this collector is not supported. It is useful for not supported platform.
cb_update	Callback to poll metrics.
cb_exit	Callback to cleanup this collector. If it is NULL, this collector is not needed to clean up.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

[N/A] Backport to latest stable release.

Configuration

[INPUT]
    Name node_exporter_metrics
    scrape_interval 1

[OUTPUT]
    Name null

Debug/Valgrind output

$ valgrind --leak-check=full bin/fluent-bit -c a.conf 
==147192== Memcheck, a memory error detector
==147192== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==147192== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==147192== Command: bin/fluent-bit -c a.conf
==147192== 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/10/22 09:49:14] [ info] [fluent bit] version=2.2.0, commit=a9a2eb1c59, pid=147192
[2023/10/22 09:49:14] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/10/22 09:49:14] [ info] [cmetrics] version=0.6.4
[2023/10/22 09:49:14] [ info] [ctraces ] version=0.3.1
[2023/10/22 09:49:14] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[2023/10/22 09:49:14] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[2023/10/22 09:49:14] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /proc
[2023/10/22 09:49:14] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /sys
[2023/10/22 09:49:14] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[2023/10/22 09:49:14] [ info] [output:null:null.0] worker #0 started
[2023/10/22 09:49:14] [ info] [sp] stream processor started
^C[2023/10/22 09:49:17] [engine] caught signal (SIGINT)
[2023/10/22 09:49:17] [ warn] [engine] service will shutdown in max 5 seconds
[2023/10/22 09:49:18] [ info] [engine] service has stopped (0 pending tasks)
[2023/10/22 09:49:18] [ info] [output:null:null.0] thread worker #0 stopping...
[2023/10/22 09:49:18] [ info] [output:null:null.0] thread worker #0 stopped
==147192== 
==147192== HEAP SUMMARY:
==147192==     in use at exit: 0 bytes in 0 blocks
==147192==   total heap usage: 47,205 allocs, 47,205 frees, 7,832,452 bytes allocated
==147192== 
==147192== All heap blocks were freed -- no leaks are possible
==147192== 
==147192== For lists of detected and suppressed errors, rerun with: -s
==147192== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

nokute78 · 2023-10-22T01:38:56Z

How to add new collector

Create common source
Create common header
Create source/header for specific platform
Add mk_list_add
Define config map
Add source to CMakeLists.txt

1. Create common source

e.g.
ne_new.c:

#ifdef __linux__
#include "ne_new_linux.c"
#elif __APPLE__
#include "ne_new_darwin.c"
#else
/* Not supported */
struct flb_ne_collector new_collector = {
    .name = "new",
    .cb_init = NULL,
    .cb_update = NULL,
    .cb_exit = NULL
};

#endif

2. Create common header

#ifndef FLB_IN_NE_NEW_H
#define FLB_IN_NE_NEW_H

#include "ne.h"

extern struct flb_ne_collector new_collector;

#endif

4. Add mk_list_add

ne.c:

    mk_list_add(&systemd_collector._head, &ctx->collectors);
    mk_list_add(&processes_collector._head, &ctx->collectors);

    mk_list_add(&new_collector._head, &ctx->collectors); // add here !

    mk_list_foreach(head, &ctx->collectors) {
        coll = mk_list_entry(head, struct flb_ne_collector, _head);
        collectors_common_init(coll);
    }

5. Define config map

Property name should be collector.%name%.scrape_interval . e.g. collector.new.scrape_interval.

ne.c:

    {
     FLB_CONFIG_MAP_TIME, "collector.new.scrape_interval", "0",
     0, FLB_FALSE, 0,
     "scrape interval to collect new metrics from the node."
    },

Add source to CMakeLists.txt

Modify plugins/in_node_exporter_metrics/CMakeLists.txt to add ne_new.c.

Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>

cosmo0920

It would be really good to have! I love this patch.
I confirmed that this patch does not have any side effect on macOS. Everything is working as expected.

nokute78 requested review from edsiper, fujimotos, koleini and leonardo-albertovich as code owners October 22, 2023 01:30

github-actions bot added the docs-required label Oct 22, 2023

nokute78 temporarily deployed to pr October 22, 2023 01:31 — with GitHub Actions Inactive

nokute78 force-pushed the in_ne_pluggable branch from 85aa10b to 38fa888 Compare October 22, 2023 01:46

nokute78 temporarily deployed to pr October 22, 2023 01:47 — with GitHub Actions Inactive

nokute78 force-pushed the in_ne_pluggable branch from 38fa888 to 17dc64e Compare October 22, 2023 02:08

nokute78 temporarily deployed to pr October 22, 2023 02:08 — with GitHub Actions Inactive

in_node_exporter: more pluggable structure

a150551

Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>

nokute78 force-pushed the in_ne_pluggable branch from 17dc64e to a150551 Compare October 22, 2023 02:30

nokute78 temporarily deployed to pr October 22, 2023 02:30 — with GitHub Actions Inactive

nokute78 temporarily deployed to pr October 22, 2023 02:56 — with GitHub Actions Inactive

nokute78 temporarily deployed to pr October 22, 2023 06:04 — with GitHub Actions Inactive

nokute78 temporarily deployed to pr October 22, 2023 06:30 — with GitHub Actions Inactive

cosmo0920 reviewed Oct 23, 2023

View reviewed changes

pwhelan mentioned this pull request Oct 23, 2023

in_node_exporter_metrics: add support for thermal_zone. #7522

Merged

7 tasks

edsiper merged commit def4ac2 into fluent:master Nov 4, 2023

nokute78 deleted the in_ne_pluggable branch November 5, 2023 09:17

BrewTestBot mentioned this pull request Nov 9, 2023

fluent-bit 2.2.0 Homebrew/homebrew-core#153758

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

in_node_exporter: more pluggable structure#8077

in_node_exporter: more pluggable structure#8077
edsiper merged 1 commit intofluent:masterfrom
nokute78:in_ne_pluggable

nokute78 commented Oct 22, 2023 •

edited

Loading

Uh oh!

nokute78 commented Oct 22, 2023 •

edited

Loading

Uh oh!

cosmo0920 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nokute78 commented Oct 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

struct flb_ne_collector

Configuration

Debug/Valgrind output

Uh oh!

nokute78 commented Oct 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to add new collector

1. Create common source

2. Create common header

4. Add mk_list_add

5. Define config map

Uh oh!

cosmo0920 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nokute78 commented Oct 22, 2023 •

edited

Loading

nokute78 commented Oct 22, 2023 •

edited

Loading